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• 'DEVELOPING A DEFENSIBLE TEST OF TEACHERS' LANGUAGE SKILLS* 

•^^ Janet S. Rose W. James Popham 

Charleston County School District . UCLA and lOX Assessment Associates 

The focus on educational accountability currently is setting the stage for 
a reaffirmation of the need for teacher evaluation as well as a shift in its 
primary intent. As Harris (1981) points out, annual escalation inper-p^pil 
expenditures, coupled with progressive declines in student achievement levels, 
has alarmed the public and forced politicians to scrutinize institutions 
responsible for providing publicly- financed education. States and local school 
-districts first addressed the need for more stringent academic standards, for 
students by implementing competency testing programs.^ One by one, states 
formalized their programs through legislative mandates until the momentum 
accelerated such that, today, nearly all states have confronted and acted upon 
this issue. . ' ^ 

'Now, we_afe beginning to see what' appears to be another domino effect. In 
order to assure quality of instruction which will give i^udents .a fair opportun- 
ity to attain the new academic requirements, the public more recently is demand- 
ing that educators guarantee the competency of the professional school staff. 

■ ' .... 

On the state level (e.g.. South Carolina) policies are being developed which 
strengthen certification requirements for beginning teachers. At the local 
level parents are asking that ineffective teachers be removed from the classroom. 
In response. School Boards are supporting stronger teacher evaluation policies 
to Hasten the removal of incompetent teachers. Such is the case in 
Charleston, South Carolina. \ 

*A symposium presentation at the annual meeting of the National -Council on 
Measurement in Education, New Orlean,s, April 23-27, 1984. 



The political forces operating within this context is changin^toe course 
of teacher evaluation (Armiger, 1981). At one time, its main piF5^^ 
formative in nature, i.e., to encourage professional growth b>^^gHj CT 

areas . for improvement, developing an improvement plan or "contracHffiM^^ 

* ^ ^ .J 

ing plan implementation and assessing the extent of improvement. Wh^ 

management-i-by-objectives approach or a clinical supervision model was follawed, 

evaluation typically occurred within a supportive and non->threateningv;environ- 

' ' ' • • ^ /' Jfr 

ment (Iwanicki, 1981); the evaluator, often the principal, was percel^^i^^ as a 



friend. As a result of political pressures, the focus of teacher evaluation 
began to shift toward more summative purposes. Administrators began to design 
teacher evaluation programs which would provide them with information they need 
tp make eii5)loyment decisions regarding promotion and tenure, dismissal, etc. 

^In ^response to school board" action, teachers became hostile and complained 
about the competencies of school board members and the unethical nature of their 
goals, (McNeil, 1981). Teachers insisted that they were being victimized and 
blamed for. Students' academic deficiencies when, in fact, changes in values, 
family structure and the cultural milieu were partly responsible. Feeling 
stripped of their professional rights, teachers strongly voiced their continued 
preference for an evaluation system built qpon trxist and aimed at self-growth 
and self-in^rovement. Indeed, teachers expressing this viewpoint present a 
logically, legally, and ethically compelling argument. They criticize school 
administrators for (1) failing to evaluate beginning teachers thoroughly and 
apj)ropriately according to cbrrent district policies and (2) allowing incompe- 
tent teachers to continue their employment with the system, tf dismissed, 
teachers who have a reasonable expectation of continiied employment, defined*" 
either explicitly (e.g., by tenure laws) or implicitly^ (e.g-/by continuing 
contract laws) can claim infringement ojf a property interest and insist upon 
their pirocedural due process guarantees. Because expectations of . contract ^ 



renewal presume competence, the courts have rightfully placed the burden of 

^ '■ ^ ■ ■. ) ■ . ' 

proof for incompetence upon school authorities. From an ethical ^perspective, 

* • . ■• 

terminating a twenty-year vetbran teacher for ^corap^tence ds difficult to^ 
justify without blatant and unquestionable documentation.. 

Evaluators, mostly school principals, are placed in an uncomfortable 

> 

conflict-of-interest situation. Once in a supportive role -^upervisors who 

guided teachers towards professional growth — they are now accused of shirking 

their responsibilities and, as a consequence, are forced by policy to become a 

critic and potential enemy of the teacher with whom they must work. Can a 

principal reconcile the helping and supportive function with appraisal of 

teacher coiDpetence for summative decision-making? What will happen^ to- the 

climate and rapport so vital for positive job morale? 

Within this tense and strained atmosphere, stronger teacher evalxiation 

policies and new evaluation programs began to emerge. States and local school 

districts, like Charleston, were pressured by the public to implement immediately 

a program to identify incompetent teachers and either remediate or remove them.. 

On occasion, new teacher evaluation policies were initiated as a reaction to an 

explosive "incident" vfhich drew public attention to a, need fo* stronger evalixa- 

tion practices. Such an incident occurred. in the Charleston County^ School Dist- 

,, , ' ' , ■ ^ 

rict (CCSD), a district of approximately 42,00a students and 2,200 teachers, 

located ori the coast of South Carolina. . 
Scene I, The Crime \ 

The tale begins with a ratheA seemingly insignificant, but not uncommon, in- 
" cident. A note written by a teacher containing unforgiveable grammatical errors 
was sent home to a parent who, horrified at the teacher's basic skill deficien- \ 
cies, turned the document over to a member of the Charleston County School 
Board of Trustees. The B&ard member's reaction was not expected, given the 
general climate of educational accountability sweeping the i>ation and the 



school district at the time. He was horrified and shared his horror^ with the 
media who shared it with the public." Suddenly a great deal of , interest was 
generated in teacher evaluation. -'The School Board presumed that current teach- 
er evaluation practices were inadequate and did not do the job intended if < 
teachers' deficiencies, such as the brie exposed in this inci'dent, had not; been 
corrected. In their discussion of teacher evaluation in Charleston Countyj 

board menfcers asked whether skili' ^deficiencies displayed in the teacher's note 

- (' 
*were common to other teachers. Several members felt that it was necessary to 

test teachers' level of basic skills in order to assess the extent of the 

problem. - - 

♦ 

Scene II, The Solution 

A subcommittee of the School Board drafted a proposal for a new teacher 
evaluation program to replace the "weak" clinixajr-s^ervision model which re- 
lied, for the most part, on principals' subjective ratings of teachers. ' Among 
^the provisions of the teacher evaluation program proposed^by the School Board 
were a "professional" test and a "writing exercise." Although the policy under 
consideration stated that evaluation data would be used to identify and correct 
existing areas of need, it was evide^it that a primary purpose was to. weed out 
"illiterate"' teachers. • 
Scene III, The Reaction 

Fear, anxiety and anger spread throughout the school district. - Teachers 
reacted in a predictably negative fashion to the Board's proposal. They per- 
ceived the new program, especially the basic skills component, as an unabashed 
insult. Slowly, the relationship between the School Board and thfe district's 
teachers • deteriorated. 
Scene IV, The Road to Compromise 

The District Superintendent, placed in an obviously uncomfortable situation', 
^gathered together members of his cabinet to discuss the teacher evaluation and 



testing requirements and to prepare a statement of their reactions which woulU 
be given to the School Board. ' The. cabinet recommended that a test, ileveloped ^ 
by a reputable company, be adminigtered to ^teachers. Those who "fail" would 
be evaluated further to identify needsjso that appropriate remediation activi- 
ties could be provided. Thus cabinet members acceded to the Board?s de^sire for 
an "objective',' method of identifying below-standard teachers but maintained a , 
focus on the formative features of the evaliiation system. No mention was made * 
of dismissing teachers on the Basis pf. their performance on a test. Teachers, 
however, did not agree with the position of the cabinet. 

Differences in viewpoints between School Board members and district 
administrators, coupled with the' growing hostility of those involved, necessi^, 
tated action by the -Superintendent. , He created an ad hoc conjdttee of a^dmin- 
istrators, principals, and teachers to review the Schpol Board's proposal and 
make recommendations. The committee .was divirleci into subcoitlmittees which would- 
thoroughly investigate various aspects of the proposal . I^e CCSD Director of 
Evaluation and Research (the first author) chaired the siS)Committee on "Testing 
and Writing Skills" which studied the advantages and disadvantages of testing, 
as weli as alternative methods of evaluating teachers' -skills; This 
subcommittee recomanended use of teachers' lesson plans to assess basic writing 
needs-; This proposal had t4ie advant.age of Evaluating writing skills and other 
teacher competencies that were more closely associated with actual classroom , 
instruction. In addition, this approach would be l^ss expensive, would target 
remedial needs, and would be less apt to ^dampen,, teacher morale. The 
subcommittee also recommended that,, if the School Board insisted upon a writing 
test, it be administered' only to those- teachers for whom there was cause. , ' 

After considering the input of the ad hoc committee, the School Board 
approved a teacher evaluation policy in June ,,^1982. The policy required that 
the state's, performance evaluation instrument, "Assessment of Performance in 



Teaching" (APT) .developed to certify beginning teachers; would be'usedon all 
experienced teachers as well. Teachers' lesson planS-would be revi&w^jd^ and a 
brief writing sample on a suitable topic' would be obtained. In* dddition, those 
teachers who did i^t make an acceptable score on th^APlV would -be' required to 
tJike a' basic skills test. Test results would be used to identify areas of 
deficiency so that, they could be remediate'd. Although *riot stated openly, the 
original intent of the policy — to weed out inconqpetent teachers — remained a 
primary objective. v 

The selection or development of a basic skills test was assigned to the 
Evalusttion and^Research (E^R) office, along with a three-month deadline. E§R 
sta^f voiced serious . concerns about , the legal inplications of 'the proposed 
teacher evaluation policy, as well as the timeline and technical aspects of the 
assessment tools used to implement the policy. Having been given this charge, 
E5R took the position that a sensible and legally de'^nsible program could be 
designed as long as pfertinent legal issues were considered and sufficient time 
was allowed to develop appropriate' instrumentation. In response, the School 
Board accused the administration of espousing slow-down- rhetoric in order to 
delay program inplementation. • 

Recognizing the lack of understanding of testing issues displayed by some 
administrators and 'School Board members, the E5R director prepared a brief 
report describing legal issues and technical concerns, then organized a 



meeting to discuss the report. The E5R director invited some leading experts 
n the area o:^ tests and^glsurement to attend the, meeting as consultants. 
These expjerts/ were urged to share their reacM<nis and recommendations. The 
meeting concluded' by giving the E5R office two years and sufficient funding 
to develop a legally defensible and technically sound test to assess teachers' 
basic skills. The responsibility for the teacher assessment policy^ s iraplen 
mentation was left to the Superintendent and School Board.. - > 



Political and Legal Consideratidns 

In planning a teacher evaluation program focused, in part, on the assess- 
ment of teachers' basic skills, one must be attentive to the legal issues 
that arise when a basic. skills test is used for summative decision-making. 
Although inconroetence is indeed a legitimate c^use for .terminating a teacher, 
as' Strike and Bull (1981) note, granting tenure (or qpntinuing employment 
status) presumes competence which-^^must be disproven with rather persuasive 
evidence. In addition, if teacher evaluation practices result in the fermina- 
tion of-a-4i^propoltionate number of teachers from a legally protected minority, 
the/ school district must be able to defend the procedures in court. A strong 
defe^\se requires demonstration that the evaluation insti::uments are valid or 
related to on-the-job performance. Two court decisions highlight the impor- 
tance of ensuring the validity of a test used for employment decisions. In 
Chance v. Board of Examiners (1971) and Walston v. County School Board oj 
Nansemond (1974) termination practices were found to violate- en5)loyees' rights 
because the educational agencies failed to demonstrate a relationship between 
content of the evaluation instrument and successful job performance. 

Although the use of tests for teacher termination. has not yet reached the 
^ ^ federal courts, the district's administrators were attentive to the counsel of 
experts in the field (e.^., Trachtenberg, 1981) who recommend that school 
districts use tests developed from a formal job analysis. of the teachex^pro- 
fession. This viewpoint led to an earlier rejection of the Board's recom- 
mendation to" use the high schoollevel of - the Comprehensive Test of Basic 
' ^ Skills , a test given' to students. Also, developers of the South Carolina's 
Education Entrance E^camination (EEE) :and the National Teacher Examinations' 
(NTE) Core Examxould not permit CCSD to use these tests for purposes other 
than "those for which they werte designed,^ . ' 
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The administration also stuSied the federal government's 1978 Uniform 
Guidelines on Employee Selection Procedures which described the qualities which 
. must be incorporated into a test so that it is usei equitably in making; employ- 
ment decisions. The .Uniform Guidelines state that content validity can be \ 
demonstrated if the test' s , content "is a representative| sample of the coi\tent 
of the job" or "is a necessary prerequisite to' successful job performance." 

The first decision to be made concerned the areas which the test should 
cover. At first, the School Board wanted the test td assess reading, writing, 
and mathematics. Yet, although all teachers shouH be expected to possess basic 
communication skills regardless .of what they teach, not all teachers need 
mathematical proficiency in order to perform credibly in the classroom. After 
much deliberation, the only mathematical skill that' the staff cotild identify , 
whidh was common to all teachers was "averaging two digit numbers." The ^ 
School Board reluctantly accepted the staff's recommendation to exclude mathe- 
matics from the test content. ^ ^ ^ ^ 

The next decision- concerned validity, that is, the relatio^hip of test 
content to job performance. Even though the test was intended to be a basic 

skills test iii reading and writing, not all teachers (such as chemistry teach- 

* 

ers) would* consider-^ their chief instructional mission as promoting reading and 
writing skills. It seemed more appropriate to defend the test on the grounds 
that it measured what constituted a "necessary prerequisite" for satisfactory 
performance as a teacher than to defend it as a representative sample of 
required job content. Thus, reasonable command of the ability to read and 
write was considered a necessary but not sufficient condition -for successful 
teaching^performance. This notion was to underly the identification of the ^ 
skills to be measured by tjie test. 

At this point a Request for Proposals (RFP) to develop the Teacher's Test 
of Language Skills (TTLS) was issued by the E5R office. The- contract was ^ 

^ . - 10 



awarded to lOX Assessment As^otiates (lOX^and planning begaA immediately. 
Although CCSD and lOX agreed on the majority *of activities to be inclilded in 
the plan, our respective situations and prior experiences gave us differing 
perspectives on a few key issues.' The first dssue to arise c^oncerfied, the^ 
appropriateness of the different concepts of content validity for this parti- 
cular test. CCSD was satisfied with the concept of job-relevancy described 
earlier. lOX, however, wanted to proceed one step .further and suggested that' 
we consider gathering data regarding an aspect of test quality in which the 
courts have recently shown interest, the degree of match between the skills 
assessed and the preparation examinees have previously received at teacher train^ 
ing institutions. - lOX based their interpretations upon their experiences with 
the Debra P. (1982] case in which the courts ordered that lase of the Florida 
high school graduation test was not valid unless students had been adequately 
prepared for the 'test. A 1977 "federal court decision in South Carolina approv- 
ing* use of the NTE was based upon the same logic. However, CCSfTbelieved that ^ 
similarity between test content and preparation should not be a concern, because 
(a) preparation of teachers was the responsibility of teacher education prograins 
and (b) students — the group being protected in this instance — shouldn't have 
to suffer if teacher education programs had produced teachers who were in- 
coiq)etent in basic skill^s. The district's concern focused upon skills* needed 
by teachers to perform successfully in the classroom, regardless of the quality 
of training they received. Besides, the di^rict's evaluation effort in itself^ 
revealed skepticism of the preparation their teachers had received in college. 
Accordingly, no effort was made to ascertain wjehther the contents of the TTLS* 
had been treated in CCSQ teachers' college' programs. 

isolation of Competencies to be Tested ^ • . ^ 

Although the CCSD School Board had approved a policy calling for the use 
of a tejst to evaluate teachers' reading writing proficiencies, the intent 



of, the policy anft- the interests of board members had^ to be' clar^^fied. before ^ 
•* • / ^ Skills could be ide^ified. Clarification occurred thtDugh informal convers 

tions with board membeors. By probing .^t was learned tKat board memb^s' c 
perceptions wer^ that only "literate" individuals should serve a^ public school 
^• teachers., Thusx^e natrfire ^f the test to 'be developed shoiSfld allow non- literate 
iridividuais to be ideftti^^^* B^ard members djd not want a test, which assessed . 
literacy at the lowest ;^osXible^eve.i. Rather, - the" Board defined literacy as a • 
level of proficiency c6nsotiant\.with the requirements ^f "^succes^filil performance 



as a teacher. Board me'mbers supplied insifghts which woiiJLd be helpful in the 

■ ' '■ * ■ Y •■ ^ ■ 

subsequent prepatation of test specifications. The Board believed that teaph- 
. ers should be able (1) read a variety^ of wi^it^ten mat;erials related to their 
profession and (2) write coherent and grainml[j:ically .coTtect communications to 

parents and s.tudents. ' , > 

* ■ - >. 

S 'The next step Was to conceptua^Jize; more speqj.fically, the nature of the 

^ • * ' ■ '< ■ 

skills to be ^.solatfed. It Was^ clear that, the reading and writing skills 

. s ^ V, ■ 

selected for, the test should be basic literacy skills atid, in addition, * 



unquestionably relevant to a teacher's responsibilities.- Content validity 
.'would be ensured by evaluating' a teacher's reading or writing competency within 

: ■ ' / ■ ' . • ■ ' ■ ' ^ 

the context of siraulate4^'job relevant situations, e.g., selections from 
professional journal?, te'^cher's guides, district policy manuals, etc. ^ 
Rulings of previous coUrt cases suggested an additional requirement. In the 
past, courts have insisted that teachers, be given sufficient time and assis-. 
tance/to correct deficiences before adverse emplq^ent decisions are made. ' 
Therefore, the skills selected, for the test had to be^ such that, having been 
provided with relevant remedial instijiction, teachers had a reasonable chance 
to master the skills. ^ ' • . 

^\ Given these grourM rulfes, a plan Kad to be devised which addressed key 
legal and political considerations. First, the plsi^ ha,d to yield "moderately" 
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difficult job-relevant skills for which . remediation activities could be designed. 
Second, to av^oid future legal challenges, key groups of individuals had to 
participate JLn the identification process. ^lOX and CCSD staff agreed that 
. groups affected bV decisions made on the basis of the TTLS (e.g. , teacliers) 
should have a voice in the- skiil-identification process. Also,, inclusion of 

district administrators and faculty members from South Carolina teacher edu- 

. ' • i 

cation programs would increase credibility of the skill-selection process. And,^ 
finally, perspectives from parents and community represeijtatives would gamer 
local support for the new teacher evaluation program. 

The skill-identification procedures originally proposed in the RFP weire re- 
vised following conversatipns during which lOX staff shared their experiences 
in similar ventures with CCSD staff • The initial RFP-stipulated plan required 
a screening and consolidation process. Individuals from school and community 
groups would generate or react to a list of possible reading and writing skills. 
The contractor was then to coalesce the preferences of the diverse groups into 
broader skill domains. lOX st^ff pointed out the possibility that such an 
aggregation process could result in a set of skills which would not necessarily 
coincide with the requirements described earlier, e.g., job relevancy. An 
agreement was reached to have lOX propose an initial, tentative set of skills 
which would then be reviewed by the various groups involved. 

In order to accumulate input from concerned constitutencies in a way 
that was both manageable and timely, the skill-identification process was 
designed in two stages: (1) a preliminary identification of skills and 
(2) a refinement of the skills. In stage one a sample of approximately 70 
school district administrators, principals, teachers, PTA chairpersons, con- 
stituent board members (CCSD's school governance consists of a single 
county-wide board plus a number of smaller, constituent boards), and repre- 
sentatives of key community groups completed a survey in which they reacted 

^ ' 11 
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to a small number of skills carefully prepared by lOX. A revised ver^sion of 
these skills was then shared with all certificated- staff in the district who 
thus had an opportunity to rate the importance of the skills during the second 
stage of the skills-identification process/ 

A Teacher Evaluation Committee (TEC) was creatisd to review survey f 017ns 
prior to distribution and to makS recommendations based ^on tabulation of 
survey results. The TEC, chaired by the Superintendent, consisted of board 
members, district administrators, principals, teachers, an.d faculty members 
from teacher training institutions. * In all decision-making situations, the 
Superintendent acted upon the comanittee's recommendations and took responsi- 
bility for * the^finai recommendations to be submitted to the School Board. 

The initial survey consisted of a list of skills, written as close to final 
form as possible, which were to be evaluated by respondents according to the 
importance of each skill. Skill statements were followed by a list oi 
job-relevant content apt to be used to assess the extent to which examinees 
had mastered the skills. Respondents were asked to offer sugge^ions |or 
additions, deletions, or modifications to the skills and likely content. 

The survey was completed by 70 Charleston educators and citizens- Nearly 
all respondents (approximately 91-94 percent) wished to retain all reading and 
writing skills. A few respondents recommended an additional writing skill 
based upon a new Board policy that had been approved two weeks earlier. The 
policy stated that all students-' written work turned in to teachers would be 
graded according to grammar and composition as well as appropriate content. 
Suggested TTLS items consonant with this new policy would assess teachers' 
ability to correct errors in students' compos il^^ons. Respondents also suggested 
revisions to the lists of likely content. 

The Teacher Evaluation Committee studied the report summarizing results 
from this initial review of skills. All committee membets agreed with the pro- 



J 
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posed revisions. A few members, including the Superintendent, wanted to add 
another skill — Completing Forms and Applications — which they contended the 
teachers had to do on a rather frequent basis. ^ 

In the next stage of the skills-identification process, three grdjups of 



certificated personnel', i.e., teachers, school-site administrators, ani| dis- 
trict staff,, were given an opportunity to react anonymously to the revised 
list of skills. Prior to distributing the survey, all relevant personnel were 
-formally notified that they would be given this^'skills-review opportunity. 
Surveys were distributed to principals who organized a aystem in the schools 
whereby teachers could voluntarily complete the survey. Respondents received 
a description of the skills, then rated the skills on a continuum from 1 
(unimportant) to 5 (essential) . Respondents also were encouraged to offer 

suggestions. Responses were secured from 561 individuals, that is, 25 percent 

ft 

of the certificated staff employed in the school district. Average ratings of 
each skill ranged from 3.6 to 4.7 for the three groups of respondents with 
teachers' ratings being considerably lower for each skill. The writing skills 
were rated somewhat higher than the reading skills. The percentage of individ- 
uals selecting each response alternative iridi-c^ated that a majority of individ- 
uals from each group selected ratings of 4 or 5^ for each reading and writing 
skill. • j 

The TEC reviewed the report prepared by^lOX which summarized the survey* s 
results and contained recommendations. lOX recommended deletion of three skillj 
one of which was "Correcting Student Writing." lOX staff believed that such a 
skill would be hard to justify (in court) as being necessary for all teachers. 
Although all teachers were required (by the new board policy) to correct stu- 
dents V written work, lOX questioned whether all teachers would have an opportun- 
ity to correct students' assignments on a consistent basis. The committee, 
however, believed that only one skill, "Completing Forms and Applications," 
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should be ^eleted. The rationale for this decision was based upon a re-anaiysis 
of the nature of thfe skill, rather than survey data. The coiranittee decided tiiat 
this skill relied more on reading ability and following directions than on 
writing ^ability. Also, it was not thought to be relevant to the instructional 
process* The Superintendent then presented the final version of the skills to 
be tested by the TTLS to the Board^ of ^Trustees for approval. The skills 
(described in Appendix A) included five reading skills - Details, Main Idea, 
Inference, Reference Usage, and .Analysis of Job-Related Literature - and five 
writing skills - Mechanics, Word Usage, Sentence Formation, Organization, and 
Correcting Student Writing, The reading skills would be assessed by 
multiple- choice items. The first four writing skills would be evaluated via 
a sample of teachers' writing; the last writing skill would be tested by givipg 
teachers a saBlple of student writing with errors, then asking the teachers to 
make corrections. 
Creation of Test Specifications 

A strategy similar to the one used in the skills-identification process was 
applied to the development of test specifications. Whereas CCSD initially 
wanted local educators to draft the contents of the specifications, lOX proposed 
a procedure which offered more guidance and structure without sacrificing local 
control. Given the controversial nature of the TTLS and the manner with which 
it would be used, the psychometric and legal defensibility of the instrument 
had to remain of paramount importance. - lOX had to ensure that this attribute 
of the test was not coiiqpromised in favor of the preferences of local educators. 
The approach that was finally adopted allowed local educators to have general 
initial input into draft specifications and to review all subsequent versions. 
It was stipulated, howevex^, that content suggested by local educators would be 
incorporated into the specifications only if it did not jeopardize the de- 
fensibility of the test. With care lOX and CCSD planned the activities for 
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this phase of the project, 

Reading and writing Teacher Evaluation Area Committees (TEACsO functioned 
in the suggesting-raonitoring. capacity recomme^jide4-'iJy lOX, The TEACs, composed 
of scShool district and college/university teieheir education experts in readiiig 
and writing; were involved in ^ecifi cations -development over a $ix-weeks J 
period. For each skill selectte^ti^or the TTLS, lOX prepared a document, for- 
matted according to the components used in preparing specifications, which 
included ^'possible" skills and sample items, followed by discussi6n issues re-^ 
garding stimulus attributes, response attributes, scoring procedures ^d ^ad- 
ministration procedures. The discussion issues, prepared in -the form of ques- 
tions, highlighted key decision-points to be addressed by the TEACs. The 
issues covered such topics as readability, passage length, sentence length, 
word level limitations, stimulus content and scoring rtibrics for the writing 
items. After having received the specifications, discussion issues, the TEACs 
attended a meeting moderated by an lOX staff member (the second author) during 
which they discussed item types and considered the disciassion questions for 
each skill. Following the meeting, lOX prepared tentative test specifications 
to be presented to the. TEACs for review at a second meeting conducted one 
month later. Several minor modifications were incorporated into the specifica- 
tions as a Jesuit of TEAC review. Yet, it was not until test items were written 
and reviewed that certain aspects of the specifications which needed revision 
were brought to CCSD^s attention. 
The Test Items 

Once sufficient test items had been prepared by lOX, they were relayed to 
CCSD for review by the two TEACs. Careful analysis of the items made it possibl 
to identify not only particular items that needed to be revised or deleted, but 
also ^'retroactive** modifications, mostly minor, that needed to be made on the 
TTLS test specifications. 
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When CCSD had approved all items, a field-test of the items was carried 
out in Charleston on. November 16, ,1983. In all, 938 teachers participated in 
,the field- test. Based on the responses of the field- test participants, using 



both Rasch procedures and conventional item analyses, two equivalent forms of 
the TTLS reading test were prep^ii^d by.IOX. Those test forms were submitted 
to CCSD in ^rch, 1984. * ' • ' - 

Because the vgriting. portion of the CCSD involvedf constructed, (as opposed 
to selected) responses, \t was believed imp^ortant by both CCSD and lOX personnel 
that the scQring of th^ field-test l^apers, that is, the writing samples and the 
"correcting" assignments, be done jointly. ^Accordingly, in December, 1983, two 
representatives of CCSD met with seven lOX staff members in Los Angeles to score 
air of the field-test writing reisponses. There were approximately 900 wr^ng 
samples (10 prompts) and 900 correcting responses (10 prompts). Following^j|ie 
scoring of the writing responses, lOX relayed to CCSD the summary data plias ^ 
several equidifficult prompts for both writing tasks. 
Long Distance Collaboration 

As was noted several times earlier in this analysis, the frequent inter- 
play between CCSD and lOX staff reflects a somewhat typical working relationship 
between school district personnel and an external contractor. Original plans-, 
both by CCSD as well as lOX, were substantially modified or totally scrapped 
as a consequence of letters, conference telephone calls, or face-to-face 
meetings between lOX and CCSD representatives. 

Several of the modifications originated from insights acquired by the 
collaborators; others were based upon logistical consideocations or recommenda- 
tiohs made by the District Superintendent. In situations where we felt the 
Superintendent's recommendations could jeopardize the legitimacy of the test, 
implications of his position were discussed with him. This procedure ensured 
that the Superintendent understood relevant issues and, as chief defense witness. 
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could justify his decision in a court of law. Luckily, CCSD staff were 
fortunate to have a Superintendent who has a background in testing and was 
acutely aware of political issues. * j 

Given the sensitive' nature of the test being developed, both CCSD and lOX 
staff displayed frequent symptoms of "warranted paranoia" « during the project. 
Members of the project staff , ^whether in Charleston or Los ^Angeles, constantly 
conteirplated that "day in court" when a teacher (or teachers' organization) 
would challenge the legitimacy of the TTlS as an instrument which could lead 
to. teacher termination. Both lOX and CCSD staff recognized that the^TTLS, 
given the high likelihood of litigation regarding the test ,and its use, had to 
be above reproach, both psychometrically and legally. The quest was for a test 
of teachers' language skills so patently defensible that it could withstand the 
expected legal assault. , ^ 

EyH^ry step of the test-deyelopment enterprise had to be conducted openly 
with the involvement of concerned constituencies. Every important decision, 
had to be documented with supporting .evidence. Just as fear often galvanizes 
one's effort, the prospect of a legal calamity with the TTLS spurred both CCSD 
and lOX to car^ out the test development endeavor with consummate care. 

Happily, during the entire test deveJLopment extravahganza, CCSD and lOX 
staff were able to function as true collaborators. This was not a project 
characterized as "us" against "them." Perhaps the prospect of "a , courtroom 
catastrophe stimulated the various individuals involved to function as co- 
workers in the truest sense of that word. Whether or not the threat of 
judicial jeopardy created the collaborative spirit is not clear; 'Hopefully, 
that spirit was simply a function of the individuals involved and their common 
perspectives regarding the nature of the test development task. The prospect 
of peril, however, surely was an incentive to effective collaboration. - 
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_ APPENDIX A / 

TEACHER'S TEST OF READING SKILLS ^ ' ■ 

Details > Given a^selectiori from job- re levant materials, the teacher 
can locate important details in a selection. 

Likely Content : Teachers' guides for student textbooks, 
student textbook^ professional journals, curriculum 
guides, memor^da from administrators, policy statements 
^ from school boards, adminis^tration manuals far standardized 
• tests, college-level textbooks dealing with curriculum and 
instruction, emergency instructions (such as for fire drills), 
educationally focused sections of news magazines and newspapers, 
instructional materials catafogues. * 

. \^ . - " ^ . - -^^ - ' 

Main Idea- Given a selection from job-relevant materials, the teacher 
can identify a statement that 'best represents the main idea of the 
selection. 

Likely Content : Teachers' guides for student textbooks, 
student textbooks, professional journals, curriculum guides, 
memoranda from administrators, policy statements from school 
boards, administration manuals for standardized tests, 
college-level textbooks dealing with curriculum and instruction, 
emergency instructions (such as for fire drills), educationally 
focused sections, of Sews magazines and newspapers, instructional 
materials catalogues. 

Inference. Given a job-relevant selection from which a reasonable 
inference (for example, a conclusion or a prediction) can be drawn, 
the teacher can identify a statement which best represents that 
inference . . 

Likely Content : Teachers' guides for student textbooks, 
student textbooks, professional journals, curriculum guides, 
memoranda from administrators, policy statements from school 
boards, administration manuals for stan^iardized tests^ 
college-level textbooks dealing with curriculum and instruction, 
emergency instructions (such as for fire drills), educationally 
fociised sections of rt^wS magazines and newspapers, instructional 
materials catalogues, district policies and procedures manuals 
or related documents. . 



Reference Usage . Given an excerpt from a reference tool typically 
en?)loyed by educators, the teacher can use that excerpt to locate 
needed information. 

Likely Content : Class schedules, library card catalogues, 
indexes, tables of contents, dictionaries, curriculum guides. 
Readers' Giiide to Periodical Literature, thesauri, BSAP Testing 
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and Teaching Guides, district policies and procedures manuals 
or related documents. ^ 



5. Analysis of Job-Related Literature. Given an excerpt from a publica- 
tion apt to be used by educators in their work, the teacher can identify 
(a) a statement of fact, (b) a statement of opinion, or (c) a logically 
fallacious statement. 

i 

Likely Content : Journals and magazines of subject-matter 
ass ociations • g • * /National Council of Teachers of English), 
educationally relev^it- magazines (e.g., Psycholo^ Today) , 
publicatioiljp of teachers organizations (e.g. , teachers union 
brochures), college-level textbooks ^dealing with curriculum 
arid instruction. 

. TEACHER' S TEST OF WRITING SKILLS 

^^"■"^^^^^"^^ ^'— ' ^ ^^^■^^— ^^^^ 

1. Mechanics, . The teacher can display correct use of spelling, 
capitalization, and punctuation. 

Likely Content : Letters to parents, memoranda to school 
administrators, letters of recommendation, notes to students, 
letters to community organizations, announcements for school 
bulletins. 

r 

2. Word Usage . The teacher can display correct and effective us of words 
for a given context. 

Likely Content : Letters to parents, memoranda tooschool 
administrators, letters of recommendation, notes to students, 
letters to community organizations, announcements for school 
bulletins. 



3. Sentence Formation . The teacher can display correct construction of 
sinple, compound, and conplexv. sentences. 

Likely Content : Letters to parents, memoranda to. school 
administrators, letters of recommendation, notes to students, 
letters to community organizations, announcements for school 
bulletins.' 



4. Organization. The teacher can display effective structuring of 
elements in a communication. 

Likely. Content . Letters to parents, memoranda to school 
administrators, letters of recommendation, notes to students/ 
letters to community organizations, announcements for school 
bulletins. 
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Critiquing Student Writing. The teacher can detect errors in student 
compositions'! * » 

Likely Content ;. A real or fictitious writing saseple containing 
errors in mechanics, word us^ge, sentence formation, and organi-^ 
ation. 




21 

23 



